chore(ci): temporarily disable NVSkills pipeline - #1302

The skill drifted from the code because the NVSkills CI gate blocked any PR
touching top-level `skills/`, so several evaluator changes landed with their
docs updated and the skill left behind. That gate is gone as of #1302, so this
catches the skill up.

Stored tasks (#1071, #566). `TaskInput` now carries a runner-discriminated
`spec`, and `EvaluatorTaskDefinition` has the grader-only `reference` field.
The skill still showed the flat pre-#1071 shape and told readers that held-out
ground truth required an inline `AgentEvalTaskInput` -- which would cost them
tasksets and revision pinning for a limitation that no longer exists. Three
places said it; all three are corrected.

Local execution (#1262). The skill lumped `client.evaluator.run()` together
with the `nemo evaluator ... run` CLI verb as "being retired", but only the CLI
verb still exists -- the method was removed a week ago. SKILL.md now warns about
the CLI verb alone: naming a method that cannot be called, four lines from the
seven live `.run()` calls the skill teaches (`AgentEvaluator().run`,
`Evaluator().run_sync`), invited the wrong generalization. The removal is
recorded in `troubleshooting.md` instead, which is symptom-indexed and so only
reached by someone who already called it from memory.

`GymRunnerTarget` was also missing from SKILL.md's platform-target list,
alongside the same omission in the agent-evaluation reference.

Taskset submission (#1367). `submit` grew a second shape -- `tasks` + `target`
against a live runner -- which was previously CLI-only and went out with no
skill or docs coverage. Added to the interface table and the agent-evaluation
reference, along with `GymRunnerTarget` in the target table, the four row-only
options the taskset path refuses, and the Gym-only translation limit.

The returned `AgentEvaluatorJobResource` deliberately has no `get_result()` or
`download_artifacts()`, while every other job example in the skill ends in
`get_result()`. That trap gets its own troubleshooting row.

`evals.json` graded the agent on producing `nemo evaluator evaluate run
--spec`, the very path SKILL.md says not to build on. Both verbs take
identical spec flags, so the eval was rewarding the discouraged one for no
benefit.

Deliberately NOT included: the skill updates written for #1173. That PR closed
unmerged, so `Evaluator.run_dataset_sync` and
`client.evaluator.evaluate_dataset` do not exist. `evaluate_dataset` on main is
the *backend* contract method, which makes the rename look landed when it is
not. The public surface is still `run_sync` and `submit(metric=..., config=...)`.

Every claim was verified by executing it against main rather than reading the
source, which caught two errors in my own first draft: an example missing the
required `resources_server`, and a claim that `env_vars` can hold a callable.
It cannot -- it is `dict[str, str]`, so pydantic refuses one at construction
and it never reaches the serializability guard. Only `hydra_params` is
`dict[str, Any]`. (The `_gym_target` docstring names both and is likewise
overstated, but that is merged code and out of scope here.)

Four tests added, each mutation-verified. The largest gap they close is that
`store_resources` -- the skill's canonical stored-task example -- was only ever
asserted as text, so no schema change to `TaskInput` could fail it. It now runs
against the real resource signatures and re-validates through the wire form
`create` actually posts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Sandy Chapman <schapman@nvidia.com>

The skill drifted from the code because the NVSkills CI gate blocked any PR
touching top-level `skills/`, so several evaluator changes landed with their
docs updated and the skill left behind. That gate is gone as of #1302, so this
catches the skill up.

Stored tasks (#1071, #566). `TaskInput` now carries a runner-discriminated
`spec`, and `EvaluatorTaskDefinition` has the grader-only `reference` field.
The skill still showed the flat pre-#1071 shape and told readers that held-out
ground truth required an inline `AgentEvalTaskInput` -- which would cost them
tasksets and revision pinning for a limitation that no longer exists. Three
places said it; all three are corrected.

Local execution (#1262). The skill lumped `client.evaluator.run()` together
with the `nemo evaluator ... run` CLI verb as "being retired", but only the CLI
verb still exists -- the method was removed a week ago. SKILL.md now warns about
the CLI verb alone: naming a method that cannot be called, four lines from the
seven live `.run()` calls the skill teaches (`AgentEvaluator().run`,
`Evaluator().run_sync`), invited the wrong generalization. The removal is
recorded in `troubleshooting.md` instead, which is symptom-indexed and so only
reached by someone who already called it from memory.

`GymRunnerTarget` was also missing from SKILL.md's platform-target list,
alongside the same omission in the agent-evaluation reference.

Taskset submission (#1367). `submit` grew a second shape -- `tasks` + `target`
against a live runner -- which was previously CLI-only and went out with no
skill or docs coverage. Added to the interface table and the agent-evaluation
reference, along with `GymRunnerTarget` in the target table, the four row-only
options the taskset path refuses, and the Gym-only translation limit.

The returned `AgentEvaluatorJobResource` deliberately has no `get_result()` or
`download_artifacts()`, while every other job example in the skill ends in
`get_result()`. That trap gets its own troubleshooting row.

`evals.json` graded the agent on producing `nemo evaluator evaluate run
--spec`, the very path SKILL.md says not to build on. Both verbs take
identical spec flags, so the eval was rewarding the discouraged one for no
benefit.

Deliberately NOT included: the skill updates written for #1173. That PR closed
unmerged, so `Evaluator.run_dataset_sync` and
`client.evaluator.evaluate_dataset` do not exist. `evaluate_dataset` on main is
the *backend* contract method, which makes the rename look landed when it is
not. The public surface is still `run_sync` and `submit(metric=..., config=...)`.

Every claim was verified by executing it against main rather than reading the
source, which caught two errors in my own first draft: an example missing the
required `resources_server`, and a claim that `env_vars` can hold a callable.
It cannot -- it is `dict[str, str]`, so pydantic refuses one at construction
and it never reaches the serializability guard. Only `hydra_params` is
`dict[str, Any]`. (The `_gym_target` docstring names both and is likewise
overstated, but that is merged code and out of scope here.)

Four tests added, each mutation-verified. The largest gap they close is that
`store_resources` -- the skill's canonical stored-task example -- was only ever
asserted as text, so no schema change to `TaskInput` could fail it. It now runs
against the real resource signatures and re-validates through the wire form
`create` actually posts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Sandy Chapman <schapman@nvidia.com>

The skill drifted from the code because the NVSkills CI gate blocked any PR
touching top-level `skills/`, so several evaluator changes landed with their
docs updated and the skill left behind. That gate is gone as of #1302, so this
catches the skill up.

Stored tasks (#1071, #566). `TaskInput` now carries a runner-discriminated
`spec`, and `EvaluatorTaskDefinition` has the grader-only `reference` field.
The skill still showed the flat pre-#1071 shape and told readers that held-out
ground truth required an inline `AgentEvalTaskInput` -- which would cost them
tasksets and revision pinning for a limitation that no longer exists. Three
places said it; all three are corrected.

Local execution (#1262). The skill lumped `client.evaluator.run()` together
with the `nemo evaluator ... run` CLI verb as "being retired", but only the CLI
verb still exists -- the method was removed a week ago. SKILL.md now warns about
the CLI verb alone: naming a method that cannot be called, four lines from the
seven live `.run()` calls the skill teaches (`AgentEvaluator().run`,
`Evaluator().run_sync`), invited the wrong generalization. The removal is
recorded in `troubleshooting.md` instead, which is symptom-indexed and so only
reached by someone who already called it from memory.

`GymRunnerTarget` was also missing from SKILL.md's platform-target list,
alongside the same omission in the agent-evaluation reference.

Taskset submission (#1367). `submit` grew a second shape -- `tasks` + `target`
against a live runner -- which was previously CLI-only and went out with no
skill or docs coverage. Added to the interface table and the agent-evaluation
reference, along with `GymRunnerTarget` in the target table, the four row-only
options the taskset path refuses, and the Gym-only translation limit.

The returned `AgentEvaluatorJobResource` deliberately has no `get_result()` or
`download_artifacts()`, while every other job example in the skill ends in
`get_result()`. That trap gets its own troubleshooting row.

`evals.json` graded the agent on producing `nemo evaluator evaluate run
--spec`, the very path SKILL.md says not to build on. Both verbs take
identical spec flags, so the eval was rewarding the discouraged one for no
benefit.

Deliberately NOT included: the skill updates written for #1173. That PR closed
unmerged, so `Evaluator.run_dataset_sync` and
`client.evaluator.evaluate_dataset` do not exist. `evaluate_dataset` on main is
the *backend* contract method, which makes the rename look landed when it is
not. The public surface is still `run_sync` and `submit(metric=..., config=...)`.

Every claim was verified by executing it against main rather than reading the
source, which caught two errors in my own first draft: an example missing the
required `resources_server`, and a claim that `env_vars` can hold a callable.
It cannot -- it is `dict[str, str]`, so pydantic refuses one at construction
and it never reaches the serializability guard. Only `hydra_params` is
`dict[str, Any]`. (The `_gym_target` docstring names both and is likewise
overstated, but that is merged code and out of scope here.)

Four tests added, each mutation-verified. The largest gap they close is that
`store_resources` -- the skill's canonical stored-task example -- was only ever
asserted as text, so no schema change to `TaskInput` could fail it. It now runs
against the real resource signatures and re-validates through the wire form
`create` actually posts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Sandy Chapman <schapman@nvidia.com>

The skill drifted from the code because the NVSkills CI gate blocked any PR
touching top-level `skills/`, so several evaluator changes landed with their
docs updated and the skill left behind. That gate is gone as of #1302, so this
catches the skill up.

Stored tasks (#1071, #566). `TaskInput` now carries a runner-discriminated
`spec`, and `EvaluatorTaskDefinition` has the grader-only `reference` field.
The skill still showed the flat pre-#1071 shape and told readers that held-out
ground truth required an inline `AgentEvalTaskInput` -- which would cost them
tasksets and revision pinning for a limitation that no longer exists. Three
places said it; all three are corrected.

Local execution (#1262). The skill lumped `client.evaluator.run()` together
with the `nemo evaluator ... run` CLI verb as "being retired", but only the CLI
verb still exists -- the method was removed a week ago. SKILL.md now warns about
the CLI verb alone: naming a method that cannot be called, four lines from the
seven live `.run()` calls the skill teaches (`AgentEvaluator().run`,
`Evaluator().run_sync`), invited the wrong generalization. The removal is
recorded in `troubleshooting.md` instead, which is symptom-indexed and so only
reached by someone who already called it from memory.

`GymRunnerTarget` was also missing from SKILL.md's platform-target list,
alongside the same omission in the agent-evaluation reference.

Taskset submission (#1367). `submit` grew a second shape -- `tasks` + `target`
against a live runner -- which was previously CLI-only and went out with no
skill or docs coverage. Added to the interface table and the agent-evaluation
reference, along with `GymRunnerTarget` in the target table, the four row-only
options the taskset path refuses, and the Gym-only translation limit.

The returned `AgentEvaluatorJobResource` deliberately has no `get_result()` or
`download_artifacts()`, while every other job example in the skill ends in
`get_result()`. That trap gets its own troubleshooting row.

`evals.json` graded the agent on producing `nemo evaluator evaluate run
--spec`, the very path SKILL.md says not to build on. Both verbs take
identical spec flags, so the eval was rewarding the discouraged one for no
benefit.

Deliberately NOT included: the skill updates written for #1173. That PR closed
unmerged, so `Evaluator.run_dataset_sync` and
`client.evaluator.evaluate_dataset` do not exist. `evaluate_dataset` on main is
the *backend* contract method, which makes the rename look landed when it is
not. The public surface is still `run_sync` and `submit(metric=..., config=...)`.

Every claim was verified by executing it against main rather than reading the
source, which caught two errors in my own first draft: an example missing the
required `resources_server`, and a claim that `env_vars` can hold a callable.
It cannot -- it is `dict[str, str]`, so pydantic refuses one at construction
and it never reaches the serializability guard. Only `hydra_params` is
`dict[str, Any]`. (The `_gym_target` docstring names both and is likewise
overstated, but that is merged code and out of scope here.)

Four tests added, each mutation-verified. The largest gap they close is that
`store_resources` -- the skill's canonical stored-task example -- was only ever
asserted as text, so no schema change to `TaskInput` could fail it. It now runs
against the real resource signatures and re-validates through the wire form
`create` actually posts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Sandy Chapman <schapman@nvidia.com>

The skill drifted from the code because the NVSkills CI gate blocked any PR
touching top-level `skills/`, so several evaluator changes landed with their
docs updated and the skill left behind. That gate is gone as of #1302, so this
catches the skill up.

Stored tasks (#1071, #566). `TaskInput` now carries a runner-discriminated
`spec`, and `EvaluatorTaskDefinition` has the grader-only `reference` field.
The skill still showed the flat pre-#1071 shape and told readers that held-out
ground truth required an inline `AgentEvalTaskInput` -- which would cost them
tasksets and revision pinning for a limitation that no longer exists. Three
places said it; all three are corrected.

Local execution (#1262). The skill lumped `client.evaluator.run()` together
with the `nemo evaluator ... run` CLI verb as "being retired", but only the CLI
verb still exists -- the method was removed a week ago. SKILL.md now warns about
the CLI verb alone: naming a method that cannot be called, four lines from the
seven live `.run()` calls the skill teaches (`AgentEvaluator().run`,
`Evaluator().run_sync`), invited the wrong generalization. The removal is
recorded in `troubleshooting.md` instead, which is symptom-indexed and so only
reached by someone who already called it from memory.

`GymRunnerTarget` was also missing from SKILL.md's platform-target list,
alongside the same omission in the agent-evaluation reference.

Taskset submission (#1367). `submit` grew a second shape -- `tasks` + `target`
against a live runner -- which was previously CLI-only and went out with no
skill or docs coverage. Added to the interface table and the agent-evaluation
reference, along with `GymRunnerTarget` in the target table, the four row-only
options the taskset path refuses, and the Gym-only translation limit.

The returned `AgentEvaluatorJobResource` deliberately has no `get_result()` or
`download_artifacts()`, while every other job example in the skill ends in
`get_result()`. That trap gets its own troubleshooting row.

`evals.json` graded the agent on producing `nemo evaluator evaluate run
--spec`, the very path SKILL.md says not to build on. Both verbs take
identical spec flags, so the eval was rewarding the discouraged one for no
benefit.

Deliberately NOT included: the skill updates written for #1173. That PR closed
unmerged, so `Evaluator.run_dataset_sync` and
`client.evaluator.evaluate_dataset` do not exist. `evaluate_dataset` on main is
the *backend* contract method, which makes the rename look landed when it is
not. The public surface is still `run_sync` and `submit(metric=..., config=...)`.

Every claim was verified by executing it against main rather than reading the
source, which caught two errors in my own first draft: an example missing the
required `resources_server`, and a claim that `env_vars` can hold a callable.
It cannot -- it is `dict[str, str]`, so pydantic refuses one at construction
and it never reaches the serializability guard. Only `hydra_params` is
`dict[str, Any]`. (The `_gym_target` docstring names both and is likewise
overstated, but that is merged code and out of scope here.)

Four tests added, each mutation-verified. The largest gap they close is that
`store_resources` -- the skill's canonical stored-task example -- was only ever
asserted as text, so no schema change to `TaskInput` could fail it. It now runs
against the real resource signatures and re-validates through the wire form
`create` actually posts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Sandy Chapman <schapman@nvidia.com>

The skill drifted from the code because the NVSkills CI gate blocked any PR
touching top-level `skills/`, so several evaluator changes landed with their
docs updated and the skill left behind. That gate is gone as of #1302, so this
catches the skill up.

Stored tasks (#1071, #566). `TaskInput` now carries a runner-discriminated
`spec`, and `EvaluatorTaskDefinition` has the grader-only `reference` field.
The skill still showed the flat pre-#1071 shape and told readers that held-out
ground truth required an inline `AgentEvalTaskInput` -- which would cost them
tasksets and revision pinning for a limitation that no longer exists. Three
places said it; all three are corrected.

Local execution (#1262). The skill lumped `client.evaluator.run()` together
with the `nemo evaluator ... run` CLI verb as "being retired", but only the CLI
verb still exists -- the method was removed a week ago. SKILL.md now warns about
the CLI verb alone: naming a method that cannot be called, four lines from the
seven live `.run()` calls the skill teaches (`AgentEvaluator().run`,
`Evaluator().run_sync`), invited the wrong generalization. The removal is
recorded in `troubleshooting.md` instead, which is symptom-indexed and so only
reached by someone who already called it from memory.

`GymRunnerTarget` was also missing from SKILL.md's platform-target list,
alongside the same omission in the agent-evaluation reference.

Taskset submission (#1367). `submit` grew a second shape -- `tasks` + `target`
against a live runner -- which was previously CLI-only and went out with no
skill or docs coverage. Added to the interface table and the agent-evaluation
reference, along with `GymRunnerTarget` in the target table, the four row-only
options the taskset path refuses, and the Gym-only translation limit.

The returned `AgentEvaluatorJobResource` deliberately has no `get_result()` or
`download_artifacts()`, while every other job example in the skill ends in
`get_result()`. That trap gets its own troubleshooting row.

`evals.json` graded the agent on producing `nemo evaluator evaluate run
--spec`, the very path SKILL.md says not to build on. Both verbs take
identical spec flags, so the eval was rewarding the discouraged one for no
benefit.

Deliberately NOT included: the skill updates written for #1173. That PR closed
unmerged, so `Evaluator.run_dataset_sync` and
`client.evaluator.evaluate_dataset` do not exist. `evaluate_dataset` on main is
the *backend* contract method, which makes the rename look landed when it is
not. The public surface is still `run_sync` and `submit(metric=..., config=...)`.

Every claim was verified by executing it against main rather than reading the
source, which caught two errors in my own first draft: an example missing the
required `resources_server`, and a claim that `env_vars` can hold a callable.
It cannot -- it is `dict[str, str]`, so pydantic refuses one at construction
and it never reaches the serializability guard. Only `hydra_params` is
`dict[str, Any]`. (The `_gym_target` docstring names both and is likewise
overstated, but that is merged code and out of scope here.)

Four tests added, each mutation-verified. The largest gap they close is that
`store_resources` -- the skill's canonical stored-task example -- was only ever
asserted as text, so no schema change to `TaskInput` could fail it. It now runs
against the real resource signatures and re-validates through the wire form
`create` actually posts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Sandy Chapman <schapman@nvidia.com>

The skill drifted from the code because the NVSkills CI gate blocked any PR
touching top-level `skills/`, so several evaluator changes landed with their
docs updated and the skill left behind. That gate is gone as of NVIDIA-NeMo#1302, so this
catches the skill up.

Stored tasks (NVIDIA-NeMo#1071, NVIDIA-NeMo#566). `TaskInput` now carries a runner-discriminated
`spec`, and `EvaluatorTaskDefinition` has the grader-only `reference` field.
The skill still showed the flat pre-NVIDIA-NeMo#1071 shape and told readers that held-out
ground truth required an inline `AgentEvalTaskInput` -- which would cost them
tasksets and revision pinning for a limitation that no longer exists. Three
places said it; all three are corrected.

Local execution (NVIDIA-NeMo#1262). The skill lumped `client.evaluator.run()` together
with the `nemo evaluator ... run` CLI verb as "being retired", but only the CLI
verb still exists -- the method was removed a week ago. SKILL.md now warns about
the CLI verb alone: naming a method that cannot be called, four lines from the
seven live `.run()` calls the skill teaches (`AgentEvaluator().run`,
`Evaluator().run_sync`), invited the wrong generalization. The removal is
recorded in `troubleshooting.md` instead, which is symptom-indexed and so only
reached by someone who already called it from memory.

`GymRunnerTarget` was also missing from SKILL.md's platform-target list,
alongside the same omission in the agent-evaluation reference.

Taskset submission (NVIDIA-NeMo#1367). `submit` grew a second shape -- `tasks` + `target`
against a live runner -- which was previously CLI-only and went out with no
skill or docs coverage. Added to the interface table and the agent-evaluation
reference, along with `GymRunnerTarget` in the target table, the four row-only
options the taskset path refuses, and the Gym-only translation limit.

The returned `AgentEvaluatorJobResource` deliberately has no `get_result()` or
`download_artifacts()`, while every other job example in the skill ends in
`get_result()`. That trap gets its own troubleshooting row.

`evals.json` graded the agent on producing `nemo evaluator evaluate run
--spec`, the very path SKILL.md says not to build on. Both verbs take
identical spec flags, so the eval was rewarding the discouraged one for no
benefit.

Deliberately NOT included: the skill updates written for NVIDIA-NeMo#1173. That PR closed
unmerged, so `Evaluator.run_dataset_sync` and
`client.evaluator.evaluate_dataset` do not exist. `evaluate_dataset` on main is
the *backend* contract method, which makes the rename look landed when it is
not. The public surface is still `run_sync` and `submit(metric=..., config=...)`.

Every claim was verified by executing it against main rather than reading the
source, which caught two errors in my own first draft: an example missing the
required `resources_server`, and a claim that `env_vars` can hold a callable.
It cannot -- it is `dict[str, str]`, so pydantic refuses one at construction
and it never reaches the serializability guard. Only `hydra_params` is
`dict[str, Any]`. (The `_gym_target` docstring names both and is likewise
overstated, but that is merged code and out of scope here.)

Four tests added, each mutation-verified. The largest gap they close is that
`store_resources` -- the skill's canonical stored-task example -- was only ever
asserted as text, so no schema change to `TaskInput` could fail it. It now runs
against the real resource signatures and re-validates through the wire form
`create` actually posts.

Signed-off-by: Sandy Chapman <schapman@nvidia.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>